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ABSTRACT ^ • ■ 

The purpose of this handbook is to encourage and 
assist woaen* s studies directors and faculty to perfors or obtain 
evaluations of their proqraas in order to assist in their decision 
■aking needs, and to know how, when, and to what ends prograa 
evaluation can be used. Evaluation is defined and described. Five 
■ajor approaches are listed: professional judgient, seasuresent, 
decision-theoretic, goal-free, and the fit between goals and reality. 
Goals and objectives are defined and possible dileaaas between long- 
and short-range goals discussed. A quasi-experiiental design is 
described and exeaplified for a proqram with seven objectives. Survey 
instrument s, interviews, and observation techniques are described and 
coipared. The need for planning data analysis in advance of data 
collection is presented. The final evaluation report is discussed 
with reference to its audiences (the client and other interested 
persons) and its forsat. An example of an executive susnary report is 
appended, together with abstracts of 13 evaluations of women's 
studies programs, and a list of resource materials. (CTM) 
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NOTE TO READERS 



The spirit of this handbook is one of enlightened self-in:erest. Its pur- 
poses are to help women's studies directors and faculty realize that no 
single approach to evaluation is sacrosanct and to show how evaluation 
can serve their decisionmaking needs. In this sense, the goal is to educate 
the consumer to know how. wher>. and to what ends program evaluation 
can be used. 

The handbook provides information on: 

• Choosing among typesV evaluation and using evaluation; 

• Turning program goals into objectives; 

• Selecting appropriate research designs; 

• Deciding cn instruments and items; 

• Analyzing data; 

• Reporting results; 

• Using other materials (summaries of recent studies and resource 
materials). 

This handbook grew out of an evaluation workshop given at the 
National Women's Studies Association (NWSA) Fir-. IJational Confer- 
ence m Uwrence. Kansas, in June 1979. The three leaders of the 
workshop and the authors of this handbook are: Mary Ann Millsap. NIE 
Program on Teaching and Learning; Naida Tushnet Bagenstos. NIE Pro- 
gram on Dissemination and Improvement of Practice; and Margaret 
Talburtt, Formative Evaluation Research Associates (Ann Arbor, 
Michigan). 

Early in 1979. Dr. Elaine Reuben, coordinator of NWSA. had asked 
us to conduct the workshop because of increasing demands for evaluation 
services, growing concerns about what evaluation is. and questions about 
how It could best be used. To many, evaluation was seen as a baffling, 
threatening, yet necessary exercise. Drawing on our own experiences in 
evaluating innovative educational programs and products, we hope to 
remove some of the mystique of evaluation and provide directors with a 
useful tool for assessing their programs. The handbook reflects our 
commitment to the improvement of education for student development 
and our belief in the positive uses of evaluation toward that end. 
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Evaluation is not a mystical or unknow; ^le force but rather a pragmatic 
tool for program improvement. 

This handbook also reflects the philosophy of the NIE Testing, 
Assessment, and Evaluation Program, which seeks to expand the tech- 
nical roles of evaluators to include more interaction with stakeholders — 
with those persons having an investment in the program and its objec- 
tives. Through a variety of NIE projects, especially in urban education, 
evaluators are working directly with school, community, and parent 
groups in the design, conduct, and use of program evaluations. This 
hahdbcK>k, with its emphasis on evaluation and its uses for women's 
studies directors, is another effort to translate that philosophy into 
practice. 

We are particularly indebted to Elaine Reuben for making the 
workshop and subsequent handbook possible. To the women's studies 
directors from colleges and universities all over the country who par- 
ticipwed in the workshop, we are also indebted. They served both as a 
field test for the materials and a support group for revisions. Jane Siegel of 
the American Institutes for P search provided us with both materials and 
suggestions for conducting the workshop and preparing the handbook. 
Had she not been giving a worksl^op on evaluation for the Women's 
Educational Equity Grants Program, she would have been the fourth 
workshop leader at the NWSA conference. To the many pioneers in 
women's studies and in evaluation who reviewed earlier drafts of the 
handbook, we are grateful for their thoughtful comments on content, 
sample items, and additional references. These people include Florence 
Howe and Paul Lauter of SUNY's College at Old Westbury, Holly Knox 
of the Project on Equal Education Rights, Meredith Larson of SRI 
International, and C. B. Crump, formerly of NIE's Social Process/ 
Women's Research Program and now at the University of California at 
Berkeley. 

The handbook is a living document, designed to be revised as we 
learrt more aK)Ut women's studies and evaluation. We encourage readers 
to contribute to the content and to comment on the usefulness of the 
h::ndKx)k. Please send your suggestions to Mary Ann Millsap, Program 
on Teaching and Learning, Testing, Assessment and Evaluation Pro- 
gram, National Institute of Education, 1200 19th Street, N.W., 
Washington, D.C, 1020S, 



Mary Ann Millsap, Seniv>r Associate 
National Institute of Education 
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EVALUATION AND ITS USES 



Introduction 

Evaluation is essentially a process of assigning value, of trying to deter- 
mine the worth of st)me thing or some activity. Each of us conducts our 
own evaluations every day. To say that 1 did better or worse than 1 
expected on an exam or in giving a workshop is an evaluative statement. 
In assigning grades to students, one kx)ks at whether the students learned 
as much as or more than some expected average. One makes judgmerits 
usmg mtormation aK)ut the meeting of a criterion (or of criteria) and 
assigns grades. 

Program evaluation differs st^mewhat from such everyday evalua- 
tions. In judging a program, one tries to be explicit about what its 
i>bjectives are. what measures are going to be used to see whether the 
objectives have been met. and what factors seem to be responsible for 
creating the effects. 

Over the past few years, a mystique has developed about what 
evaluation is. We need to understand a little more about this mystique 
before we examine how evaluation can be of use to women's studies 
programs. Twenty years ago, evaluations consisted of the review of a 
program or its personnel by a panel of experts, persons whose informed 
judgment was employed to examine whether the proper range of pro- 
grams was offered; whether the work was organized in an efficient 
manner, and whether attitudes about a program were positive. The use of 
expert Ci^nsultants in site reviews ffilfills a needed function, and such 
amsultants have been used by a number of women's studies programs. 
These expert consultants are aware of what is going on in different 
programs and can provide a broader perspective than program personnel 
typically have. Their reports can offer recommendations for improve- 
ment and. in some cases, can provide a "seal of approval" for a program. 

W ithm the past 1 5 years, evaluations Have taken on a wider focus 
and incorptirated a number of new mechanisms. The newness of the 
tteld, in tact, is partially resptmsible for the evaluation mystique. When 
many of us were students in college, there were no courses in evaluation. 
Many ot the people currently in the field have no formal university 
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training in evaluation but have academic training in the social sciences 
(such as in survey research in stKiology or in experimental design in 
psychology). Even now there is often no one university department 
» where one can go to find out what evaluation is. Consequently, many 
questions about the terms used in evaluation (such as tests of signifi- 
cance, analysis of variance or covariance, true control groups, andquasi- 
experimental design) remain unanswered. A language has developed for 
which no lexicon is readily available. In addition, as courses and 
programs in evaluation have grown, people have developed models (such 
as formative evaluation, summative evaluation, the decision-theoretic 
approach to evaluation, goal-free evaluation, discrepancy evaluation), 
and these models are not always clearly described. 

In addition to confusion about models and terms of evaluation, 
other difficulties arise when one explores the ways in which evaluation is 
used or can be used. Evaluations designed to meet the specific informa- 
tion needs of sponsors may be quite narrowly fcxrused. Evaluations often 
are used, however, by individuals or interest groups concerned with ob- 
jectives that the evaluation may not have addressed. In some cases, the 
absence ot information is equated with negative information to the 
detriment ot programs. For example, while evaluations are (^ften assumed 
to be concerned only with a program s effects, evaluations also can look 
at the prcK*ess ot a program's development in order to provide diagnostic 
information to program managers. Such process evaluations need not 
address the impact of a program at all. Yet some audiences, upon dis- 
covering that program effects were not addressed, may conclude that the 
program was unsuccessful. Also, an evaluation focused entirely on out- 
comes, such as measures of student achievement, may not address 
adequately the quality of a program if no steps were taken to determine 
whether the results were directly attributable to the program and not to 
other fact(^>rs. 

It is not surprising, then, that prt)gram personnel often perceive 
evaluation as a threat; they assume that if evaluations do not show 
programs doing all the things they want to do, the programs will be 
punished by having funds withdrawn. This concern is not without foun- 
dation, and evaluations have been used to limit or reduce funding. While 
some programs have legitimately lost funding when they were not suc- 
cessful in attaining many objectives, evaluations often have been toc^ 
narrowly detined to fully explore what programs are doing. As a result, ^ 
these evaluations have presented an incomplete picture of program 
accomplishments. 

The pi^tential or perceived threat of evaluation may have a legiti- 
mate pi>litical base. Within much of elementary and secondary edu- 
cation, tor example, innovative programs but not traditional curriculums 
are evaluated. This alsc> has been the case with women s studies programs 
in colleges and universities. The traditional liberal arts curriculums 
seldom are required to state explicitly their objectives for students, yet 
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women's studies programs not only are expected to do so but also are 
judged by whether they have met those objectives. An additional burden 
often placed upon innovative programs is that they are seen as the 
panacea tor the ills of higher education. Hence, unrealistic expectations 
are formed, st)metimes by program personnel but more often by the 
institution. As a result, innovative or interdisciplinary programs fre- 
quently must contend with unrealistic objectives, competing with each 
other for funds and support, while traditional offerings continue without 
examination or question. 

In addition, unrealistic expectations can contribute to a premature 
evaluation of the program or to an evaluation based on insufficient 
evidence. While we have less information about women's studies pro- 
grams, some innovative elementary and secondary education ideas (e.g., 
educational vouchers) lost support after limited tryouts. The conclusions 
drawn from the few cases were generalized to apply to all such efforts. » 
The result was that the ideas were not tried out in a variety of settings to 
determine whether they could prove effective in a different environment. 

While one can argue with the decisionmaking process and about 
what is going to be evaluated, it seems likely in a period of retrenchment 
in higher education that women's studies programs and other inter- 
disciplinary programs are going to be evaluated. It is seemingly no longer 
a question of whether women's studies are going to be evaluated, but how 
they are going to be evaluated. Consequently, women's studies directors 
and faculty need to become Ixnh knowledgeable consumers and active 
prtKlucer^ c.f evaluation information. 

Typ«s off Evaluation 

There are five major approaches to evaluation, each with a very different 
purpose and methodology.' Each is listed with a brief description below. 
We should keep in mind that while these approaches differ, it is possible 
and often desirable to combine them in evaluating programs. 

Professional judgment The major purpose of this mode of evaluation is 
to certify the acceptability of a program as seen by an external group of 
ct)lleagues using pre-established reference pt:»ints. Examples of this type of 
evaliiatK^n include accreditation visits and other professional certifica- 
tion Of educational programs. The methodology frequently involves cur- 
ricular reviews, interviews, and a vast collection of supporting materials. 
The result is professional acceptance, rejection, or— stimetimes— a pro- 
visional or temporary rating. While often conducted by external groups, 
this kind ot evaluation can also be done through in-house review, 
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involving, for example, people from other interdisciplinary programs. 
This probably remairu the most typical and most well-known form of 
evaluation in higher education. 



MeMurement* One example of this evaluative approach is grading. 
Students' work is measured and compared, and a final evaluative assess- 
ment is made. Other examples are personality tests. Graduate Record 
Exams, the College Boards, and such institutional measures as the 
Institutional Goals Inventory. In all of these cases, the tests are pre- 
established, applied to all in an equal manner, and analyzed to look for 
spread or for differences among individuals. The object of the study is 
often the student rather than the pro^m. In fact, the central contro- 
versy about evaluation as measurement or testing is that the tests often 
are based on what other students learned in the past rather than on what 
was taught within the particular college or university. 

Decision-theoretic apprcwch. This form of evaluation attempts to 
involve all decisionmakers in the design of the evaluation. In women's 
studies, Marcia Guttentag pioneered the use of the decision-theoretic 
approach.^ All decisionmakers (e.g., students, faculty, administrators, 
funding agencies) are asked %list what they think are the objectives of 
the program, to rank the objectives numerically in order of importance 
(so one can then say, for example, that some objectives are 2 or 10 times 
as important as others), and then to estimate the chances of the objec- 
tives being accomplished (these are called prior probabilities, and a score 
of 0.5 would be a 50-50 chance while 0.9 would mean vimial certainty). 
Data then are collected on each objective and analyzed through the use 
of Bayesian statistics. The most useful part of the decision-theoretic 
approach is the deliberate involvement of all decisionmakers prior to the 
conduct of the evaluation. Several other models, such as Provus' dis- 
crepancy evaluation, also rely heavily on the early involvement of 
potential decisionmakers. ^ While time consuming and difficult, this early 
involvement can lead to the development and acceptance of more real- 
istic objectives for a program, to a more sensitive evaluation design, and 
to greater use of the evaluation data in decisionmaking. 



* MdrciJ CiUttcntag. cd.. E^'oluatum Studies Reuru' Annual, Vol. 2 (Beverly HilU; Sage Pub- 
lication. 1^77) Marcu Ciiittentaj;. Lorelei R- Brush. Alice Rom Gold. Mamie W. Mueller. Sheila 
Tobia*. and Marni CK>ldstein White. **Evaluattn{{ Women's Studies: A DtfcisK^n-Theoretic 
Approach. " Signs. V No, 4 (Chicago: University of Chtcaj^^ Press. Summer 1978). pp. 

^Malcolm Provus, "Evaluation of Ongoing Prt>Krams m the Public 55chool System/* The 68th 
VcarbtKik cj/ the Situmjl SiKvety fijT the Study of £du*.un<m. Part II (C^iCtigo: The University of 
C'hKa(^lPrrss. pp. lil-lSl 
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Goal«free evaluation. The important aspect of this form of evaluation is 
that it pays as much attention to the unintended consequences of a 
program as to intended outcomes, and every program has both. The 
intended results are those embodied by' the goals and objectives. But 
unintended consequences can be equally important. For example, the 
Career Facilitation workshops of the National Science Foundation's 
Women in Science Program are designed to provide information about 
potential careers in science to young women. One result found by 
evaluators, however, was that the development of the workshops led to 
the creation of strong and supportive networks among womtn who were 
already scientists. An evaluatio>» that searches for unintended conse- 
quences is called goal free because it is not bound by the program's goals. 
Since many positive aspects of programs— and some of the more trouble- 
some, negative aspects— are unintended, evaluation designs might, 
insofar as possible, look for all effects, goal related or not. The 
methodology of goal-free evaluation is almost anthropological. One 
would examine the artifacts of a program, talk with its living survivors, 
and review the records, all the while checking to see what happened and 
why rather than whether objectives were met. Major proponents of this 
approach are Michael Scriven and Malcolm Parlett." 

Asseument of the fit between goals and reality. This form of eval- 
uation is probably the approach that most innovative programs use. The 
purpose of the evaluation is to understand whether the stated objectives 
<.)t a program were achieved. The methodology can range from a classic 
experimental design to informal interviews. 

With the exception of goal-free evaluation, all of the models 
described above also can incorpt^rate an examination of what was accom- 
plished in light of what was expected. However, because the assessment- 
ot-tit mcxJel could in tact be an overarching one, much of the rest of the 
handKxik addresses its application to women's studies programs. 



Us*s of Evaluation 

The conduct of evaluations, particularly the last three discussed above, 
can he usetiil to people directing women's studies programs. They can 
provide mtonnation on the extent to which programs are attaining their 
object I ve.s, on where strengths and weaknesses are, and on what areas 
require improvement. Moret)ver, evaluations can buttress other kinds of 
intornicition that women'.s studies programs probably ouj^ht to be col- 
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leccing i\s well. For example, while evaluations often are concerned with 
the impact of profjrams on students, or faculty, such additional questions 
as the tollowinj^ should be posed, 

• Has there been an increased demand for women's studies programs 
(^ 8 I h*as the 'unduplicated count of people enrolled in women's stud- 
ies increased, arc more students anticipating taking women's studies 
programs).^ 

• Have the prt)grams increased the amount of external money coming 
into colleges and universities? 

• Have more students been attracted to the college or university in partf 
beCcUise ot the women's studies programs? 

• Have students enrolled in women's studies programs had higher re- 
tention rates in colleges and universities than other students? 

These are all forms ot unanticipated consequences, discussed under goal- 
tree evaluation. 

For women's studies programs to compete successfully for the allo- 
cation ot resources and staff, such information could be extraordinarily 
useful. In tact, women's studies directors should be encouraged to view 
evaluation in an active rather than a passive light. Programs frequently 
can be criticized for not having sufficient evaluation data, but if programs 
have S4>lid, positive results, the dat/ can be used not only in discussions 
with the institution but also in securing funds from other public and 
private sources. 



Critical Charactaristics of an Evaluation 

The time, money, and effort put into a program evaluation will be en- 
hanced it the complete range of alternatives is considered before the 
research begins. By linking decisionmaking needs^to research questions at 
the start, greater impact of the evaluation is ensured* The following 
discussion presents the characteristics that we feel program directors 
should consider in developing evaluatiorts. These points are particularly 
relevant in a goals/reality evaluation, but could be useful in the design of 
i)ther approaches to evaluation as well. Each of these characteristics is 
expressed as a continuum, but most evaluations choose a balance some- 
where between the two extremes. The availability of time, money, and 
start IS a necessary consideration in designing an evaluation. The more 
tormative, comprehensive, and thorough an evaluation is, the higher its 
financial, temporal, and staffing costs. 



FORMATIVE | 

^To impfove tho p.ogram) 



» SUMMATIVE 

(To judge tne impact of the program) 



A formative evaluation is meant to collect information that improves the 
pro«ranv Collection usually starts soon after the program begins and is 
imgomg. It is the type o* evaluation a program director often wants, and 
IS sometimes referred to either as a process or management evaluation. 

A summative evaluation judges the program's merit. Was it worth 
the money.'. Is it qualitatively acceptable.' Should it be continued or 
expai\ded.' Certainly these considerations are important to the program 
directors as well as to those in decisionmaking positions abtwe.them. 
, bummatiVe evaluations usually come toward the end of a program and 
generariy ^ect the fate of a program. For this reastm, summative 
evaluations often are feLtred and are confused with other types of 
evaluation. Most programs want to do some of each; the real issue is 
where to stand between the two. 















PURPOSI 






■■ 1 


DESCRIBE % 


% EXPLAIN 






(What happened) 


(Why it happened) 



Similarlv, one needs to decide what amount of the evaluirtive work will 
de$cri/v impact (e.g., 75 percent of all students answered at least 80 
pcryent ot the items correctly) and how much of the study can expbn 
impact (c.^.. students answered the questions correctly for these three 
reasons). To determine the appropriate position to take on these two 
purptvse continiia, the program director should consider such pt)ints as 
whv and fur whom the evaluation is beins done. 



SCOPK OF THI IVALUATION 



NARROW « 

(eg. Cognitive growth 
of students) 



■% COMPREHENSIVE 

(All components of the program, 
cognitive and affective) 



l\)c^ the proKram seek to have an impact on only one element, such as 
student awareness of women authors in the United Kingdom.' Or does 
the prt)Kraai hope to affect mult:->le cognitive and affective variables.' Are 
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students m clasps x tt) be ct)mpared tt) students in class y? To students in 
other schtx)ls? The mt)re ^odh a pn^sram has» the more comprehf nsive 
its evaluation should be. 

One may chtx)se to evaluate the objectives one by one or in a closely 
integrated desij^n. The greater the scope ot evaluation, the higher its 
costs, but the more likely it is tt) capture the richness t)t a program and its 
effects. Conversely, a small program with limited resources for evaluation 
would be wWl advised to tocus on a tew objectives and keep its evaluatitm 
appropriately retined. 



TIMI FRAME 

MOMENTARY « 1 LONGITUDINAL 

(One-day interview) (Across entire academic year) 



An evaluation can take place at the end of the semester, at multiple 
pt)ints durinjsj the semester, or evenMt)ng after the semester is over. By 
iiu^nitorinn a program t)ver time, one if in a much stronger position to 
understand its evolutitm and impact. The monitoring can be simple or 
complex, depending on the scope, purpose, and resources of the evaU 
uation. The program director should decide befc^re a program starts if the 
k)ngitudinal inkirmation is desirable. If data are not obtained before or as 
a program starts, one cannt>t gt> back to retrieve them after the program is 
under\\ ay, since by that time one no longer can distinguish between what 
was know n bett>re and what was learned during rhe program. 



TIME FRAME 

RETROSPECTIVE « 1 PROSPECTIVE 

(How It happened) (Where it is going) 



A retrospective evaluation seeks ro explain with hindsight why a project 
evolved as it did. This type of information often is require^! by funding 
agents who want to know why their money was spent as it was. the 
other hand, a prospective evaluation may be of greater interest to a 
program director who wants to determine options for the future. An 
evaluatu>n can Ntiidv K>th ot these issues if desired. Again, the real 
decisioii IS one ot emphasis. 



MSIQN AND IMMIMIIITAttON 

* ♦ CONOUCTIO BY iXTIRNAL PARTY 

(WmNndtrartment) (Outside fMe«rch«f») 



Betore an evaluation begins, one must know whi) is doing it. Many 
options exist. Program directors may choose to do all evaluation them- 
selves. This poses some problems in terms of credibility and sheer 
stamina. Some administrators, citing potential vested interests, may not 
believe the findings of an internal evaluation, especially one of program 
effects. In contrast, it is possible to hire someone f^om an outside firm to 
conduct all evaluation activities. Between these two points are many 
options. A program director may wish to hire an outside evaluator to 
design a study and instruments, but carry it out internally. A program 
director could hire an outside expert in the field to review parts of the 
program as appropriate (e.g. , a noted English professor to comment on 
the curriculum of "Women Authors 401"). Or, some universities may 
have in-house researchers available. 

* Selection of an evaluator is complicated by issues of political, 
economic, and professional competency. Is it better to have an evaluator 
who knows a program well or not at all.' How much money is available for 
outside rest)urces.' Who is credible to those who commissioned or man- 
dated the evaluation.' Will the evaluaior(s) be evaluated.' Regardless of 
who conducts an evaluation, a program director would do well to review 
the purpt)ses, scope, time frame, design and implementation, method- 
ology, and feedback with all candidates for the job. Do not accept an 
evaluator who rekise.s to discuss the parameters of the evaluation in plain 
English. 



METHOOOLOQY 

INSTRUMENTED « 1 OPEN-RNDEO 

(Quantified information) (Qualitative information) 



There are many different and legitimate ways to collect data.' The pur- 
po.ses ot the evaluation, the acceptability of the methodologies to those 
involved, the necessary costs and time restraints all are to be taken into 
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account in deciding; this point. Are packaged tests to be used? Would, 
interviews be preferable? Will surveys be returned if they arc sent out? 
Should they be open-ended or confined to quick checklists? Can a pro- 
gram director review or «hape questionnaires or interviews? The im- 
pi^rtance of even a small pre-test of pilot data collection cannot be 
stressed enough to determine whether the instruments really probe for 
what the evaluators want to knoW. Frequently the best methodology is a 
combination of approaches* permitting the inclusion of all types of data 
from all types of sources. 



FEEDBACK 

ORAL I » WRITTEN 

(Workshop presentations) (Formal reports) 



This is often the most ignored aspect of evaluation. The point is to 
analyze and report the findings so that .they can be used rather than 
forgotten. For some audiences, a 200'page report is a must and will be 
thoroughly utilized. But, for others, a two-page executive summary, 
highlighting the major findings, will have far more impact. Perhaps a 
workshop or I -hour oral presentation would gather more attention. A 
video tape may dcxrument the results better than a report. In choosing 
the miKle(s) of feedback, consider the learning styles of the audience and 
what they would view as credible and interesting documentation of a 
program's impact. 

Summary 

This chapter is intended to dispel some of the myths surrounding program 
evaluation and to replace them with a basic knowledge of evaluation. 
From the five general approaches to evaluation we moved into an ex- 
ploration of the parameters that need to be considered in the design of an 
evaluation. As a summary ni^te, the following questions translate many of 
the principles of evaluation into si)me very practical questions. By 
reviewing these points early in a program's life, directors will be ab*:^ to 
K^rmulate an evaluation plan that best serves their needs as well ts to 
prepare tor the summative judgments others will make. 



Pnietieal St«ps for Condiietlng IvahHitloiis 

A. Identify evaluation needs and purposes. 

Who should be involved.' 

Who should decide the scope of the evaluation? 

What process is ISest suited to making decisions.' 

B. Create research questions. 
What must be studied.' 

What additional information would be desirable? 

C. Create evaluation design. 

What kinds of evidence are necessary and credible? 
What financial and personnel resources are available? 

D. Create evaluation budget. 

How much time will be involved for various personnel? 
What salary and direct and indirect costs are involved? 

E. Pr epare evaluation working plan. 

What are the specific evaluation tasks? 

Who should do them? 

When should they be accomplished? 

F. Collect evaluation information. 

What are the sources of information? 
What ethical questions are involved? 

G. Prepare information fi)r analysis. 

What computer-related resources are necessary and available ? 

H. Conduct information analysis. 

What analytical methcxls are appropriate? 

What interpretations can be made, given the strengths and weak- 
nesses of the study? 
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1. Report the information. 

Who are the audtences? 

What are their decisionmaking needs? 

What are their learning styles? 



The remaining chapters address in more detail the practical steps to 
consider in conducting an evaluation. These include translating goals 
into objectives and research questions, creating research designs, se- 
lecting instruments and items, analyzing information, and reporting the 
results. 
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GOALS AND OBJECTIVES 



bmce evaluation Involves judgment against some standard, it is frequent- 
ly useful to judge a program in light of its own goals and objectives. As 
discussed in the tirst chapter, measuring a program in terms of its objec- 
tives is only one approach to evaluation. It is, however, an extremely 
common one and one that is generally useful to decisionmakers. In this 
chapter, the pnxess of developing statements of goals and objectives is 
described; if a program is to be judged in terms of whether the goals and 
objectives are accomplished, they must be clearly stated. The process of 
making the goals and objectives explicit can do more than guide the 
evaluation. It can help faculty and students focus on what they want to 
accon\plish and guide program development so that each component 
(e.g. , course, special event) can contribute to the desired end. 



Definitions 

Goals are generally long-term, fairly global aims. A goal statement con- 
cerning women'.s studies from the University of Massachusetts, for 
example, is: 

... the development of frameworks and methodologies which 
integrate women's experiences and scholarly disciplines, and which 
ultimately will provide the incentives and expertise necessary to 
frtect change m the larger community.' 

Objectives are usually more specific and attainable. Some eval- 
uations ak> describe them as measurables. At the very least, there must 
bo a direct way to gather evidence aK)ut their achievement. Often the 



^■\\\>nw:-, Scvidii-s (.'.o,,! St,it,fmem. ' I'mversity of MavMchuMftts. Amhrrit. M.«iMchus«tts. 
(rffcrfiKi-vl m L rvlrr^j./u.u^ W.ttuti * Siudu:^ Pn^cim hv c:jru!vn Rhi>des. Old l\>min.on 
I, nistfrMf> Ri-Nf.ir«.h K.iirui.ttKin, Novcmher h>7«. p 58) 



ERIC 



13 

2'> 



methixi tor evaluation included in the objective. Two types of objec- 
tives — impact and priKess — ^are important. As the names imply, the 
tbfmer UH)ks at results while th* latter looks at the ways used to achieve 
them. An example ot an impact objective is: 

Students in pre-service teacher education courses shall apply an 
analytical scheme tor assessing sexism in textKxiks. (Method of 
evaluation: Given a textbcx)k, 75% of the students will select and 
use correctly an appropriate analytical scheme to assess sexism.) 

And a related prcKess objective could be: 
0 

to develop and teach a section of the Foundations of Education 
course that presents methods of analyzing sexism in textbooks. 
(Method of evaluation: Analysis of course syllabus.) 

An evaluation of the program logically might show that the process 
oV»jective was achieved (the section of the course was developed and 
taught) but the impact objective was not (students couldn't analyze the 
textKx)ks for sexism). In that case, the staff and students would need to 
liK^k further to discover why the course was not successful and, as a result, 
would probably want to make program adjustments. 

D«v«loping StatonMnts off ObJ«ctiv«s 
from Statomonts off Qools 

In this section, a general goal statement is presented along with a few 
examples of objectives that are logically related to it. 

As part of the Feminist Movement, Women's Studies courses should 
function: ( 1 ) to help women examine alternative ways of Ux)king at 
their roles in society and the assumptions of our culture, and (2) to 
discover and provide new information on the history, culture and 
accomplishments of women. . . . In addition to its interdisciplinary 
nature, what makes Women's Studies different from most other 
academic subjects is that it attempt to foster artective as well as 
coiijnitivc learning.' 
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The tbllowinj? objectives could bt» derived from the goals listed above. 

Faculty shall offer a course examining sex-role stereotypes and their 
social and cultural riwts. The course shall include self-exploration 
and how stereotypes affect students and shall present means of over- 
coming them (process objective). 

Students shall be able to respond to simulated real life situations In a 
non-stereotypic manner (impact objective). 

Faculty shall develop and teach feminist focused courses in history, 
literature, stKiology, and psychology (process objective). 

Students shall be able to analyze content in sociology, history, 
literature, and psychology from a feminist perspective (impact 
objective). 

Faculty shall develop and teach an interdisciplinary course that 
traces the development of women's roles in 19th century America 
(prix;ess objective). 

Students shall be able to apply concepts from sociology, psychology, 
literature, and history in an essay about specific women in 19th 
.century America (impact objective). 

The goals and objectives cited abtwe are all framed around desirable 
outcomes for students. Women's studies programs frequently also have 
objectives that concern faculty and the institution as a whole. For 
faculty, objectives may include acknowledgment of women's studies as a 
legitimate academic discipline, acceptance by faculty that students 
should be encouraged to take women's studies courses, and realiration 
that a feminist perspective should permeate academic disciplines. For an 
institution as a whole, objectives could include coordination among the 
many t>fferings on and off campus that relate to women, such as women's 
center, or community organizations. Whatever the specific objectives of 
women's studies programs, an evaluation can address whether these 
t>bjectives have been attained and how much progress has been made in 
attaining those objectives. 

Before an evaluation can be designed to measure the achievement of 
objectives, a few additional steps are helpful. First, after stating all ob- 
jective.s, sti»ff should categorize them. The categories should reflect the 
differences between prtKess and impact objectives and also the target of 
the t>bjective. For example, there may be objectives related to student 
learning, to student attitudes, and to the impact of the program on the 
total institution. Once categories are developed, staff should choose 
among those categories and objectives .so that the program can be judged 
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in light of the most important issues. Generally^ there are too many 
objectives in a program for an evaluation to be all-inclusive. The selec- 
tion process, then, helps frame the design. 

ACavMt 

There may be a dilemma involved in designing an evaluation around 
explicit statements of goals and objectives. For women's studies, for ex- 
ample, the long-range goal may be to infuse the entire curriculum with 
feminism. As a path toward that goal, some intermediate goals — 
developing specific feminist courses — and objectives may be both neces- 
sary and desirable. Faculty and staff are more able to control and to 
measure the achievement of thfe latter goals, but, in the long run, they 
may be more concerned with the former. If the evaluation looks only at 
the feminist courses, it will not present guideposts or ideas for improving 
the more general atmosphere for women on campus. 

Porter and Eileenchild define feminism as . . the desire to in- 
crease the power and autonomy of women as individuals and as a group 
(so that) they will be able to make informed and flexible choices in their 
education and their lives."' If a women's studies program accepts this 
definition and sees itself as both catalytic for others and important in its 
own right, it must take care to be evaluated in terms of both long- and 
short-range goals. 

The problem, of course, is that the achievement of the longer term 
goals are seldom under the control of feminists themselves. Since eval- 
uation results frequently are used to cbntrol programs, it is tempting to 
omit questions that are neither controllable by the program nor apt to 
lead tcT positive answers. On the other hand, omitting such questions 
deprives feminists of what may be important information. The nature of 
dilemmas is that there are no easy answers. Here, again, program de- 
velopers and staff must decide what they want and indicate that to the 
evaluators. 



^Njncv .M. Pi^rtifr and MafXarrr T BiiernehitJ, The EZ/rtaimess of Witrngn i Studies T<fik7un^ 
^\X',whin>{tiin. V*X'. National Institute of EJucdtuin. mprcM)»p. 2. 
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RESEARCH DESIGNS 




Research designs are built around specific objectives. Often the most 
difificult step in constructing a research design is the first one— deciding 
what objectives the evaluation is going to address. An initial list of 
objectives for a program is often long, cumbersome, and totally unwieldy. 
It may contain 15 or 20 objectives. An initial list of objectives for 
women's studies programs could be equally long and unwieldy, especially 
when one considers objectives for students and graduates, for faculty and 
administration, and for the program and its impact on other efforts. The 
key. as we mentioned in the previous chapter, is to rank the objectives in 
order of priority on the basis of what objectives are the most important 
(and to whom) and what objectives must be addressed this year rather 
than next year. 

Once a list of objectives has reached a manageable size, one ex- 
amines each objective ir. turn to decide what evidence would be needed 
to ascertain the extent to which the objective was accomplished. The 
question to pose is what evidence would be the most convincing to show 
that the program is doing what one wants it to do. 

This chapter contairw two sections. The first takes a single objective 
and descnbes a design to measure its attainment. We then ask a series of 
questions about information not included in the design and build succes- 
sively more complicated designs to incorporate that information. In 
practice, one tries to anticipate these questions in advance of any data 
collecti m or analysis. The second section, building upon the first, 
presents a complete hypothetical research design for the evaluation of a 
women's studies program. 



□•sign ttnit*gi«s 

Assume that the initial objective is the following: At the end of a one- 
year introductory course sequence in women's studies, students shall have 
equitable sex role perceptions of women's roles in society. 

While the concept may be difficult to measure, the design strategy is 




quite simple. Students would be asked at the end of the year about 
sex-stereotypic views, probably through a multiple-choice instrument on 
sex role perceptions (several instruments of this type are on the market). 
This design is called a post-only design, since students are ^eried only at 
the end ot a pn)gram and only students in the. program are asked 
questions. 

Schematically, the post-only design looks like the following. 
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Someone, however, may ask: "Well, the students in the program 
probably didn't have sex-stereotypic views when they went into the 
program anyway, so how did the women's studies program make a 
difference?*' , 

To respond to that question, a somewhat different design is needed. 
Students would be asked a series of questions at the beginning of the 
program prior to any instruction and then again at the end of the pro- 
gram. The twt> sets of responses then would be compared to see whether 
there were differences in their scores. This is called a pre-post design and 
involves t)nly the students in the program. 

Schematically, the pre-post design ltx)ks like the following. 
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But once again, someone may ask: "It's fine that we know the 
students now have less steret)typic views of women's and men's roles than 
when they first enrolled in the women's studies. program. But how do we 
know that what they learned couldn't have been learned just by being on 
campus, or from their friends, or from the women's movement in 
general.'" , 

To resptmd ti> this question, a still more complex design is required, 
since we now must involve other stir its, ones not in the women's 
studies program. We need not only the pre-post design (for the beginning 
and end of the year), but also s<.)me kind of control or comparison group. 
A control group would consist of students who are identical in all respects 
to the students in the women's studies program but are not enrolled in it. 
In field experimenB, a control group is usually obtained by overrecruiting 
for a pn>gram and then randomly assigning students from thg pool to the 
program (experimental group) and to the control group. The research 
design that uses experimental and control groups is called an experimental 
design. 

Since control groups are almost imptissible to isolate in real life, 
most evaluation work these days uses comparison groups. Examples of 
comparison groups for women's studies may be a random sample of stu- 
dents fn)m the rest of the college or university or a random sample of 
students within a given department (in the event that the women's 
studies program was only open to students within a given department). 
The research design that uses experimental (women's studies students) 
and comparist>n groups is called a quasi-experimenud design. These are the 
strongest yet most realistic designs since they enable us not only to 
examine progiam effects ovxr time but also to determine whether any 
changes are the result of the program itself or stem from other factors. 

•Schematically, the quasi-experimental design kxiks like the fol- 
lowing. 
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Sci faf, we have discussed only one objective related to student 
learning. We have not yet addressed the multiple objectives of women's 
^studies programs for students, faculty, and the institution as a whole. The 
next section proposes a sample evaluation design that has multiple 
objectives. 

A Sample Co^fipr«li«nsiv« R«sMreh Design 

Let's assume that the objectives of a women's studies program have been 
detined and listed in order of priority. Assume that there are three objec- 
tives for students in the women's studies program, one program objective 
for graduates, one objective for department faculty, and one adminis- 
trative objective for ccH)rdinated work with a campus-based women's 
center. 

A description of each of the objectives and of the composite re- 
search design follow- It should be noted that the objectives have been 
selected more for their diversity than for their cohesiveness. One could 
say that the first three objectives are of prune importance to the faculty 
teaching the specihc courses, that the next three objectives are of prime 
concern to the administrative staff in the college or university, and that 
the hnal objective is a first step in the long ranj: plan of the women's 
studies cix)rdinator to infuse feminism throughour che campus. 

Over the course of the year, studeixts in the mnnen \ studies program will have 
less stereotypic tieus of womeyxs roles in society: i ^se views can be attributed 
to the program and not to i)ther facttns (impact obj ctive). 

This example was detailed in the precedin section. In short, at the 
beginning ot the year, students in the women ^ studies program and a 
comparison group of students randomly selected from the college or uni- 
versity as a whole or from the department in whi-rh the course is housed 
shall be given (under identical administrative conditions) an instrument 
on sex role perceptions. The same instrument shall be u administered to 
the same students at the end of the year. Changes in scores from the 
beginning and end of the year shall he compared tor each group, as will 
differences m scores between the two groups. 

SfiJent5 in U'cnnfTi's studies (yrograms shall have developed leadership skills and 
shall fetl more confident in ma/cmg presentations before groups (impact 
objectivt?). ' 

This IS really two objectives, orte related to developing leadership 
skills and one related to self-confidence. They should he discussed sep- 
arately and are coriihmed here only as a matter of convenience for the 
discussion. Changes in leadership skills and self-confidence are expected 
over time* ?hI there is need for a pre-post design, using only the women*s 
>tudies students. Onv could measure the attainment of this objective in a 



• number i)t ways, including having a panel of consultants rate students on 
varit>us points. These consultants would look at how the students present 
themselves betore groups at the beginning and end of the year, observe at 
different points in time how students work in small groups to organize 
tasks and to carry them out, They might ask the students themselves 
whether they feel, over the course of the year, they have developed 
leadership skills and a greater sense of self-confidence and self-assurance 
in what they do. Other ways of collecting information over two or more 
t?me pi)ints include using peer review, video tapes, and self-esteem rating 
scales. 

S(i4Jen6 m the women's studies Lourse m u^nnen in the ammunity shaU 
tnepiire resourc^^ materhils m services available to u)omen in the community, 
such as Uimxen's centers, rape crisis centers, CETA projects, abortion clinics, 
SiKiid security offices, employment agencies, und child care facilities, and shall 
itutlme uhere such services are located, ho%v they are staffed, vuhat quali" 
ficatiims exist for being eligible for services, how much services cost, and the 
niime and phime number of sinneime to call fin further information (impact 
objective). 

This is a relatively simple objective to, measure. One need look only 
at whether the objective was accotr.plished (a post only design), The 
main questions to ask in measuring the objective are whether all services 
have been described, whether the information is complete and has been 
double-checked, and whether the information has been shared with and 
' understcKxl by the other members of the class. One could question, 
hi>wever, whether this should be the complete objective. Once this set of 
useful resource material has been compiled, tme could argue that the 
i^bjective should ak) include st)me activities to make these resources 
known iMid available to the community at large. 

Over the cmrse «>/ the academtc year, there will be a growmg awarei\ess among 
studey\ts on campus of the winnen's studies program, and more students shdi 
pLm on eymMmg m women $ studies courses during the following year (impact 
objective). 

This IS ak) cm objective that requires information to be collected at 
two different points m time so that one can see whether move students 
become aware ot women*s studies programs and want to enroll in them. 
Information could be collected through a brief postcard survey of a 
random sample ot students at the beginning ot the year and of another 
random sample ot students at the end of the year. Such surveys might ask 
whether they were aware of the wt)men*s studies program (prior to re- 
ceiving the postcard describing it) and whether they are interested in 
enrolling in courses (course offerings coulu be listed on the postcard). 
The number ot responses would then be tallied and changes in per- 
centages trt>m the beginning to the end of the year could be compared. 
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Graduates of the mter(M$cit>limn mphen's studies program shall assess the 
program as very useful m thJir further educatum or in empkryment (impact 
objective). 

While the data collection design for this objective is quite straight^ 
forward (a tme-time only administration, usually of a*m^iled survey), 
K)th getting the data and interpreting the results are problematic. 
Among the first matters to decide are; how long a person must be a 
graduate of the program to be included in the sample; how to find ad^ 
dresses for these graduates S4.> they will receive the survey; how to get a 
decent respt)nse rate (that is, how to make sure most of the people return 
the survey form). One could decide, for example, to send surveys only to 
those graduates who will have been out of the program one year. One 
then could make sure forwarding addresses are collected before the stu^ 
dents graduated and could inform students that a survey of graduates 
would be undertaken, stressing ^how important it is for the program to 
find out what each of them is doing one year hence. In fact, if the last 
two steps are not taken well jp advance, it may not be possible to do a 
study of gn>duates at all. within the m?il survey or telephone interview 
or both, one could ask what the graduates currently are doing and 
whether they findihe courses they took in women's studies useful in what 
they are doing. One could also ask for examples of how the program has 
been useful, both professionally and personally. It also may be helpful to 
get anecdotal data — specific information abour specific graduates — to sejc 
whether these data match the more general information obtained 
thri>ugh the survey items. >X^ile one canm>t generalize from the anec- 
dotal information, the use of individual cases and short success stories can 
be very powerful. 

What is most problematic about measuring the attainment of this 
objective is that pursuing further education and obtaining employment 
involve many factors thairare unrelated to the degree in women's studies. 
Sit IS alsi> the case that it may take 5 to 10 years to move into a goixl 
professional poskion. Furthermore, if one hasn*t the financial resources 
for additional education or if one cannot find a job because of tight laK)r 
markets, one cannot fault the women's studies programs, even though 
some may tr>* to do that. 

The problems assiKiated with measuring the objective should not, 
however, preclude women's studies directors from conducting such sur- 
veys (we will be talking more aK)ut surveys in the next chapter). Alumni 
otficcs or career placement oftices within the college or university may 
CiH)pvrate in di>ing such a study and may help with the mechanics t)f such 
a hursr'ev. Little is known o\ the activities of women's studies graduates, 
and a Ci^ntinuing concern ot many, K>th within and outside the women's 
studies tielJ, IS that the degree hampers, or at least doesn't suppi^rt, 
c*ijeer i>ptions tor graduates. Without the data, one cannot address these 
concenvN. 



By the nui of the aeademu: year, the majmty uf faculty m the department of 
hiiuny shall reammer}d to the students they counsel that the students enroll in 
the U'omeixs studies ttmrse on winnen in I9th century American life and shall 
reammend the course he required for a departmental major (impact 
i>hjective). 

This is another case where two objectives were combined into one. 
In constructing a research design, one should treat them separately. 
These objectives are not as simple to measure as they seem. They may 
involve asking the history faculty whether they are recommending taking 
the particular course and whether there are plans afix)t to have the course 
become a required course for a» departmental major, but one also may 
want to ask students their perceptions to see if faculty counselor behavior 
IS as described. Oten it isn't. The objectives are included here not 
because they are relatively easy to measure, but because they allow us to 
raise a sign of caution. These are objectives over which the women's 
studies program may have little control or influence. In deciding whether 
to include them as objectives, program staff should consider whether they 
relate to their long-term goals and what they will do with the information 
once thev have it. 

The u-omen's stiulies cwndinutor and the director of the campus-based wmxen's 
center shall develop and implement a career and group counseling program for 
Momen students at the Mnitersit> (priKess objective). 

Ihh is a priKess objective; we are concerned with how the career 
and group counseling program develops, who is involved with its devel- 
opment, and what factors seem to have helped or impeded the develop- 
ment ot the program (including costs, staff time, space). Information 
aK)ut how the activity is proceeding is usually collected over several 
points in time in i)rder to assist the developers in creating and managing 
the activity. Such information ak) becomes useful when other activities 
ot this type are planned, since this type of evaluation (formative eval- 
u.ition) often identifies roadblocks that must beoverconie. 

Having discussed each of the seven objectives with designs for 
measuring iheir atJainment, we now can summarize the overall design for 
the evaluation of the program. The summary is presented in Figure 1 to 
assist program directors not only in planning data collection and analysis 
bur also in deciding whether one has the restnirces (both start' time and 
tunds) to conduct the evaluation. Just as there is no single type of 
evaluation, there i> no single evaluation design. Depending uptm the 
intor^iation needs ot various audiences and the rest)urces available, 
evaluation designs can K> either expanded or restricted. What is crucial 
, aK)Ut developing^ he evaluation design is the knowledge it provides 
women's studi»rdirectors prun to making commitments aK)ut what 
evaluation activities will K' undertaken. 




The following chapters provide additional information about the 
pros and cons of various methtxls of data collection, data analysis, and 
reptming. 
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ill ■ 

cno or me 
Program 


AwartnMtand 

Intomt In 
Womtn*9StudlM 

Comparison groups 
{one at b6ginnir)g, 
another at end of 
year) 


Aware of /interested 
in taking women's 
studies 
(postcard) 




Aware of/interested 
in taking women's 
studies 
(postcard) 


Impact of 
Women's Studies 
Degree 

Graduates of 
women's studies j 

i 




Perceived usefulness 
of degree, current 
activities 
(mail survey) 





Impact of Women's! 
Studies on a 
Department 

All htstory 

department 

faculty 



Increasing 
Inetnutional 
Impact (Career 
and Group Coun- 
seling Program) 

Women s studies 
director and 
women s center 
director 



Students coonseled 
to take women's 
studies, course 
becoming a require- 
ment of department 
major 

(interview) • 



Infomial interviews usually tied to the specific calendar of activity 
I about how the counseling program is developing and how road- 
; blocks to Its establishment are overcome. 
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INSTRUMENTS AND ITEMS 



In discussions of what and how important specific objectives are, it is 
uscf 1 to ask, "How are we going to know whether we have achieved 
them? What information would we and others consider believable and 
convincing?" The data collection instruments provide that information. 
This chapter is. designed to assist directors of women's studies programs in 
reviewing instruments and in knowing the questioiu^ to ask about iiutru' 
ments rather than to teach them to construct instruments themselves. 
We shall provide some sources about both available instruments and 
instrument coivstruction. 

A variety of data collection instruments can be used (from multiple 
choice surveys to structured interviews to observation). Each has its own 
advantages and disadvantages. In this chapter we shall be discussing 
various forms of data collection, concentrating on what irwtruments are 
most appropriate to use for different objectives. Before we get into a 
discussion of instruments, however, several tips on instruments in general 
may be usefol. 

One of the first questions to ask is, "What do we really need to 
know.'" One of the most common errors made in collecting data is using 
instruments that ask for information because everyone asks for it; too 
many instruments include items that never find their way into any data 
Itnalysis. 

One may at some point, for example, want to know what students' 
future plans are, whether they have decided upon major fields, whether 
they have scholarships or some other form of financial aid, what their 
marital status is, and whether they have done community organizing; but 
unless one has decided to use these data in the analysis (and has ob- 
jectives requiring that kind of information), one must question their 
appearance on an instrument. Throughout any evaluation, one must 
continue to ask, "Am ! going to use this information to improve the 
program.' Is this information going to be useful to people who are making 
decisions about the women's studies program.'" Unless one has strong 
feelings that the information is going to be useful, it may be wiser not to 
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include it. The addition of what may be superfluous items will only take 
away time from more important questions, and can lead to disgruntled 
respondents. 

How one asks for information (the method of data collection) is as 
important as the content of the specific questions. The more impress 
sionistic or affective the information sought, the more likely it should be 
collected face-to-face rather than through a survey. If aa objective relates 
to changes in behavior* it is preferable to use instruments (e.g., obser* 
vational techniques) where one can in fact examine behavior rather than 
9sk respondents about it. 

Before setting about designing instruments and items for a women's 
studies program, look at what instruments are already available. The 
creation of instruments, whether they be surveys or iriterviews or check-* 
lists, appears disarmingly simple, when in fact it is a craft (and some 
would even say an art form) requiring considerable precision. 

For example, assume one wants to fmd out w( ^ther students found a 
program or course valuable or usefiil. One could ask, "Did you find the 
women's studies program valuable? Answer yes or no." Regardless of 
what the distribution of responses is (from 100 percent yes to 100 percent 
no), how is gne going to be able to interpret the data? Someone could 
well ask, "Valuable for what.^ Compared to what? In what context? What 
was valuable — the content, the teaching style, the contact with other 
students?" Unless the instrument is designed to probe these other ques- 
tions, it is unlikely one will learn anything about the true value of the 
program. As a guide, each question should seek one defined piece of 
information about ore single item. 

There are also pitfalls to be avoided with a completely structured 
item with multiple choice answers. Take, for example, the following 
item: 

1 have found the introductory women's studies course useful in: 

1 . Deciding upon my major and in making my career plans, 

2. Finding a suppi^nt group within the college/university; 

3. Pnwiding me with examples of the contributions women have 
made in the social sciences; 

4. Making explicit the values imbedded in what is included in and 
excluded from traditional courses. 

Problems with the item include: 

• Not telling the respondent either to check all that apply or to mark the 
most important item; 

• Giving the respondent no opportunity to say that the course was not 
u.setul; 

• Combinmg two or more answers in one response (in the first response, 
one or the other or Knh may he true, hut the evaluator can't tell 
which); and 
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• HavinK a global comparison that may not be true (those who mark the 
hmrth response may not have a common frame t)f reference about what 
traditional courses are). 

sSolutions might involve asking respondents to rank responses in 
order oi impt)rtance, breaking the questions down into several smaller 
questions, permitting another response category, clarifying the broadly 
defined terms, and/or scaling responses from positive to negative. 

One can never construct the perfect instrument the first time 
around, nor is it often possible to judge in advance whether some items 
will be easily understcxxl by the respondents. Consequently, the use of 
pilot tests is very important. Pretesting or piloting an instrument can 
occur in several ways. The most common fonn is to ask a group of 
rcs|X)ndents (similar m background and experience to the respondents 
one wants in the study, but perferably not those one will query later) to 
till out the survey form or complete the interview. This is done in part to 
get responses to the items, but mostly to find out whether the questions 
are clear and sensible and whether the possible responses (on multiple 
choice items) are appropriate and complete. Such pilot tests also are used 
to find out how much time is needed to complete the instrument. The 
responses and comments then serve as the basis for revisions. 

Survey Instruments 

Survey instruments usually consist of multiple choice items, rating scales 
(the familiar five-point scales ot "always" to "never," or "strongly agree" 
to "strongly disagree"), or checklists. They may be packaged, standard- 
iced mstmments or ones constructed by the program staff or evaluators. 
They are very simple to administer and analyze (especially by computer), 
reqiure little time to complete, and provide the least amount of infor- 
mation aK)ut a program or course. Sinre all the answers are provided 
(respi)ndents check just whatever applies), one can only learn what one 
already has asked. 

Standardized instruments (available through source books) save 
development time and money, which can be considerable. Usually, 
though not always, information is provided about how gtKxl the instru- 
ments are. whether they are valid (measure what they say they measure) 
and reliable (measure consistently), what the intended respondent groups 
are, and what concepts they are trying to measure. The main concern 
with standardized itistmments is that they do not necessarily relate to 
what IS taii^fit in a program or course. Consequently, cme should examine 
re.idymade mstmments very carefully to see if they can apply to one's 
own program objectives. 

C>ne ot the mystiques about standardized tests, such as norm- 
reterenced achievement rest.s. is their aura of eternal truth, as if the tests 
measured svtme universal. It is easy to slip into giving them more credi- 
bility than they warrant. For example, in evaluating student achieve- 



ment in an alternative high school, questions were raised about whether 
the tests would be appropriate considering the school's nontraditional 
curriculum. When the student5' scores were higher than expected and 
their gains over the year much greater than those for a comparison group, 
however, the questions of appropriateness were dropped, to be replaced 
with statements about program success. 

American Institutes for Research, under contract to the Women's 
Educational Equity Act Program, has prepared a Simrcehook on Measures 
of Wcmum's Educationd Equity containing a number of survey instru- 
ments. Since many project directors and others spend a great deal of time 
and effort in developing measurement instruments to evaluate the prog- 
ress and outcon^s of their programs, this collection should ease the 
burden on project directors and on directors of women's studies programs. 

Si)me 167 instruments are described in the Sourcebook, mori than 
60 ot which may be helpful in the evaluation of women's studies pro- 
grams. The description of t*ach instrument includes measurement vari* 
i\bWs, target population, instrument type, administrative procedures, 
scoring, description of sample items, availability of instrument, and 
references. 

Measures that may be of use in women's studies programs include 
attitudes toward women's studies as an academic discipline, achievement 
motivation, fear of success, sex role stereotyping in careers, attitudes 
toward feminist issues, sex role perceptions. Title IX, career awareness, 
awareness of sexism, and attitudes toward mathematics. 

In the winter of 1979, the Sourcebook will be available through the 
Education Development Center, Inc., 55 Chapel Street, Newton, 
Massachusetts 02 160. 

Also developed by American Institutes for Research is Measures of 
Educatumal Equity for Women: A Research Monograph. It too contains 
descriptions of a number of instruments useful to directors. It can be 
obtained through American Institutes for Research, P.O. Box 1 1 13, Palo 
Alto, California 94302. 



Int«rvi«ws 

Interviews are most often used when: (1) one doesn't really know the 
range or content of all possible answers; (2) the answers may be some- 
what sensitive, and respondents are concerned about using their own 
particular wording in responding, and (3) the follow-up or probing ques- 
tions depend upon responses that — because of the complexity of the 
issues addressed — cannot be predicted accurately in advance. 

Interviews may be structured or unstructured. Structured interviews 
have pre established questions in an ordered sequence. Responses may be 
open-ended or sc^licited to fall into predetermined response categories. 
This type of interview may be the most useful when analysis time is 
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limited, when maay difterent interviewers are used, or when experience 
with interviewing is limited. Stntctured interviews are more often used 
where there is no tace-ro-tace contact (such as in telephone interviews) 
and when the interviewer is not particularly knowledgeable abt^ut the 
program. Unstnictured interviews (which customarily have very general 
questions with a variety of probes based on initial responses) are more 
commonly used in tace-to-tace situations, when the interviewer is ex- 
perienced with the interviewing process and the program, when few 
interviewers are ased, or when much time is available for careful analysis 
ot data to draw out common patterns of response. 

The format ot interview questions is quite different from multiple 
choice items, although some multiple choice items may be used in an 
interview setting, provided the form is given to the respondent to com- 
plete. The interview setting cannot be used to ask respondents to do such 
things as ''rank the following 10 items in order of imptmance," for 
respondents usually can neither retain nor reorganize that much infor- 
mation without being able to see the items written down. On the other 
hand, it erne does not have pre-set responses, the interview is quite useful 
m probing what the resp<.>ndent feels is most important. Unlike a survey 
torm, the interview is active rather than passive; the responses are gen- 
erated by the respondent rather than from among a predetermined list. 

Tliere are K)th advantages and disadvantages to the interview ap- 
proach. The advantages include freedom to respond without constraint 
and the opportunity to elicit unanticipated information. An interview 
puts a greater burden on K)th the respondent and the interviewer, since 
there are no pre-set answers and the process is usually more time con- 
suming. It overct>mes the biases the respondent may have against written 
tormij, bur assumes that respondents are verbal and articulate. The bur- 
den on the interviewer is greater as well. The quality of the interview 
depends upon how gocxl the interviewer is in establishing a sense of 
rapport and trust, and in recording all relevant information. Interviews 
are mure ditticult to analyze unless the questions, however general, are 
tramed around the main issues the evaluation is to address, which once 
again brings us back to the research design. As with survey instruments, 
interview items should tocus on those objectives that are most impt)rtant 
to the evaluation. 

An excellent^ detailed guide to all aspects of interviewing is con- 
tained in the /ntmifuer\s Mumuii put out by the Survey Research 
Center, Institute tor SiKial Research, the University of Micliigan, Ann 
ArK)r. Revised in W76 and available in a paperbound edition, it ad- 
dresses many ot the issues in interviewing and includes many helpful tips 
on the ci>nduct ot inter\'iew's. 

(.>her valuable resources tor interviews about women's studies pro- 
yranvs are those people who have been involved in reviews and eval- 
uate >ns oi Wi>men's studies programs. Nancy Porter addresses some of 
these issues m .m article. "Evaluation: Reflections of a Program Con- 



sultant/* in the Fall 1977 issue ot the W<mens Studies Newsletter. 

Fli)rence Howe has made a number ot valuable sufjgesrions about 
interviewing based upon her held work tor Seven Years Liter, a b(X)k 
t'ommissioned By the National Advi!k)ry Council on Women's Educa- 
tional Programs. She suKKest:> that it is important to interview as many 
people outside ot a program as inside, asking the same questions of 
teachers, students, and administrators, including cix>rdinators or direc- 
tors ot ethnic studies and other inter-disciplinary programs to see how the 
institution treats all deviations from traditional programs. During the 
conduct ot the interview , she found it useful to have several questions for 
each interview that the mter\ iewer knows routinely and that move from 
eliciting information aKuit rhe person (e.g., title and function at this 
college, tormal or intt)rmal relationship to the women*s studies program) 
to licneral infonnation aK)ut the program itself (e.g., its strengths and 
weaknesses, how the person would like and not like to see the program 
develop during the next five years). While large group interviews can be 
unwieldy, ^he ttnind it also useful to have small group interviews with 
tour or tive pct>ple, since such interviews often sparked a discussion, 
Hi>we eniphasi:es one of the cardinal rules of interviewing: The in- 
terviewer IS .1 collector of information and not a participant. The inter- 
viewer IS not to bring her his own ideas inti> an interview; the interviewer 
IS always to remain j neutral party encouraging tiill and complete re- 
sponses trom iiit» interviewees but is not to interject views about an issue 
or program. Lastly. Howe noted that the most difficult and yet essential 
intormiition to gather aK)ut a program is its history, not only to learn 
aK)iit rhe development of a program but also to understand the social and 
{Political dynamics ot the individuals in the program. If at all possible, this 
intormdtiori should be obtained in advance or at the beginning of the 
data cv>l lection so that \\ tonns a base against which to measure other 
reNpon>e> and aid> in the de\ elopment of probing ouestions. 

As i> also rhe case with surveys, it is otten ditticult to know in 
advance all ot rhe quc>r!ons thar should be asked in an interview setting. 
! hus, some pilot tc>rir^^ or pretesting of rhe interview schedule is 
recommended, nor only ro clarity questions and estimate response time 
bur also ro ser\ e as a device tor inrer\'iewer rraining. 

Observation 

This meth^Kl ot date*, collection deri\es from the ethnographic techniques 
ot aiuhropologv aiui can be \-er>' helptui m those situations where one is 
concerned more wirh behavioral change than wirh aftecnve or ct>grutive 
le.irnin)^. It is aUo vcr\ u>etul m loo.king at unanticipated consequences, 
Mncc observation (.an be used without having a predefiiK^d ran^e of 
acrion> in mind. In tlu>se cases where observation is being used to -^ee 
wliether a >pecitic hehavionl objective has been accomplished, one 
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needs to ask whether the observers are observing the same "thing and 
whether they have been given adequate instructions about what they are 
to look tor. We all have selective perceptions, so what may be important 
to one person may be missed by another. The greatest strengths of ob- 
servational techniques are also their greatest weaknesses. While an 
observer provides an independent, detached view, one must try to make 
sure that observers focus on the phenomena of importance. Observers 
should be well briefed on what one is trying to accomplish. In this case, 
checklists of items to look for can be very helpful to observers, especially 
if they are then encouraged to explore other areas as well. 

For example, if two objectives are to have students develop better 
leadership skills and to become more self-confident in making presen- 
tations before groups, then observers should know what leadership skills 
the program director has in rtiind and what signs of self-confidence seem 
appropriate. Suppose a class is divided into small working groups with 
instructiims to complete a particular project. An observer could then see: 
(1) how the leader is chosen; (2) how the leader behaves; and (3) how 
tasks are allotted to group members. Since the objective has an explicit 
rime sequence, observers would be used at several diflferent points to track 
development. In examining self-confidence, observers would concentrate 
as much on presentation of self as on content of materials presented. 

The ethnographic literature* including the growing field of anth- 
ropology in education, could provide to women's studies directors some 
useful guides to observer selection and training. As with other approaches 
to data collection, there is no single approach to observation, and the 
literature is abundant with observational approaches tailor-made to par- 
ticular situations and objectives. Stephen Wilson s article on "The Use of 
Ethnographic Techniques in Educational Research," published in the 
Spring 1977 issue of the Rewic of Educatiimd Research, is a very gcxxJ 
introductory piece on the rationale behind and processes of this 
approach. 

What is most important to remember about instruments — about the 
methixls of data collectitm — is that they are the source of information 
aK)Ut whether one has accomplished what one is trying to do. Instru- 
ments must be carefully examined to see whether they address all the 
areas of interest, include all the groups one is concerned with (e.g., 
students, faculty, administrators), and are technically si)und. In addition, 
instruments (from surveys to observation guides) should not be admin- 
istered until one has a gcxxl idea aK)ut how one is going to analyze and 
present the information derived from them. The next chapter addresses 
the maji)r areas of diua analysis. 

i 



)3 



5 

DATA ANALYSIS 



As with the chapter on instruments, the purpose of this chapter is to 
Aquaint women's studies directors with the different types of data anal- 
ysis» primarily so that they may become better equipped to discuss 
analysis issues with others. 

Data analysis is concerned with reiducing the bulk of information 
obtained through all the instruments into manageable form. It allows one 
to look for patterns within a group of respondents and across groups of 
respondents and, with some kinds of data, to determine whether sig- 
nificant relationships exist. 

A data analysis plan should be developed before any data Collection 
takes place. Sf/h\\e it is not always possible to be precise about all the 
analvscs^to be performed, it is crucial to have a general plan of analysis 
tied to the objectives, and to know how much time and money are 
available for analysis. Access to any computer Services also must be 
determined in advance. 

Whatever the research design selected and the kinds of instruments 
used, a variety of data analysis techniques exist. Data analysis is often the 
most baffling part of an evaluation, in part because theories of probability 
and statistical viKabularies are introduced. There is, however, nothing 
mystical about data analysis. Ever>' data analysis technique has an English 
lan^'uage equivalent. 

In fact, much of the dat.) analysis appropriate to evaluating inno- 
vative programs, including women's studies, does nor involve the use of 
the more complicated statistical techniques, since the assumptions upon 
which these statistics rest are often not relevant to real world programs. 

<On the following pages, the range of data analysis possibilities are 
illt^rated. One objective, for example, may be to tie the women's studies 
program more closely to sc^me of the institution's goals. In discussions 
with administrator^, one may have learned that the institution is in- 
terested in increasing the enrollments of nontraditional students (such as 
women in their midthirties who are returning to college after their chil- 
dren are in school). One may find that, from 1977 to 1979, enrollments 
of these women in the college/university increased from 5 to 10 percent 
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i)t the total enrollment. By Lk>kinK m the enrollment of stude 
women's studies proKrams over the same periiKl, one may find the en- 
rollment ot these women increased from 10 to 30 percent (that is, JO 
percent ot the^e Women were in women's studies courses in 1979), Stated 
another way. while the enrollment of returning women doubled in the 
past three years, their enrollment in women's studies programs tripled. 
One ak) may tind. tKrou^h a postcard sur\>ey of students throughout the 
collejje or university, that half of the returning women are very interested 
m taking a women's studies course. By usin^ these percentages — more 
tormally called descriptive statistics — tme can present a case to the ad- 
ministration that the women's studies program is an impt)rtant attraction 
to these students and assists the institution's fjoals, 

With a number ot the sample objectives used in the preceding 
chapters, thg data analysis relies on descriptive statistics (such as per- 
centages). These objectives include students* yrowin^ awareness of the 
women's studies program, an increase in the number of students planning 
to enroll m women's studies courses the following year, and the majority 
ot faculty m a department recommending student* enroll in a women's 
studies C'.uirse. l>ata from graduates on the usefulness of the program in 
their turther education or employment also can be analyzed using de- 
scriptive statistics. 

Much interview and observational data are analyzed using mucl\,the 
same approach as a journalistic investigator. Assume that one objective is 
to have a course on women in English literature become a required course 
tor a departmental major. Part of the analysis regarding the extent to 
which the i>bjective has been reached ci>uld read as follows. 



Halt ot the faculty m the English department, the faculty person 
teaching the course on women in English literature, the chair ot the 
department, the women\s studies Cix>ftJinator, and the dean of the 
collcjt^c were all interviewed separately aK)ut the prospects of the 
couTNe becoming a required course for a departmental major. While 
rfie coordiniiti>r ^md faculty member teaching the course felt the 
course would complement the existing requirements, the faculty 
queried thiUight it superfluous. Yet when asked whether they had 
reviewed the reading list, talked with the faculty member teaching 
the course, and discussed the course cimtent with any students, ali , 
res{\)nded no. The chair, with the concurrence of the dean, has 
secured stable funding for the course but was reluctant to pursue 
reqi:irin^ it, tearing it would lead to a pn^liferation of/required 
courses. 




This excerpt in evNenti.illv a ciMidensation of a series of interviews with 
coiKerned p<irtieN obtained through a structured interview tormat fol- 
lowed bv v)me probing questions. There is ni^thm^ quantitative aKmt 



the iU^alyM>, hut it is data analysis nonetheless, 

The more quantitative data analysis usually' appears in ettorts to 
assess chat^yes in student attitudes or learning. The tollowin^ ohjective, 
jjited tn C :iiapter 5, tor example, requires statistical analysis. 

«.Ver the course ot the yeai^-^iidents m the women's studies 
projjram will have less stereotypic views ot women's roles in society, 
and these views t an he attrihuted to the program and not ro other 
factors. 

Here oeu- wiuild prohahly use one ot the availahle standardized insrru- 
meiUs in which numerical scores are derived trom a set ot items usin^ a 
"strongly ayree" to "strongly disagree" scale. Statistical techniques then 
wiHild he -used to V\\. at the distrihiition ot sciires to see whether dif- 
ferences amonu the scores tor the two time periods (pre and post) at)d the 
two groups (women's studies studentAnd the comparison ^roup) could 
he Kised on chance variation or whether the scores follow a sufficiently 
set pattern so that differences are most prohahly not due to chance. 

For specifics on what techniques should he used in wUit situations, 
refer to materials listed in Appendix C:.- Resource Materials. Two recent 
puhhcarions mav he quite useful since they present step-hy-step statistical 
analysis. The U'fcfc'A \i\a\xux\um H.»ui/x).)/c, prepared hy the American 
Institutes for Research, i.s designed tor individuals who want tq suhmit 
proposals to the \X'omen's hducational Equity Act and need" help in 
developing adequate evaluation plans, and tor those who already have 
projects hut need help in conducting evaluations. While the Hurui/^oofe 
does not speciticallv addres.s «oals and ohjectives of women's studies 
programs, the chapter on input assessment (how questimis affect the 
designs of evaluation) and tl\i> chapter on statistical methods (presented 
in a step-hv-step fashion) may he quite helpful. The ei«ht-volume 
/V(.^nam hi.i/iun.m K\i hv Lvnn Lyo-ns Morns, Carol Taylor ht:-C)ihKMi, 
and Ntarlene h. Henerson - although geared tor uso ui school districts-— 
mav he helpful as well. The statistieal analysis volume is one ot , the 
ciearesr m tnllow. The kit, m Millie volumes or as a complete set, is 
ay.iilahle tn.ni S,ii,v rnhlications, inc., P.O. \\^\ Beverly Hi'lis, 

C ihforni.i "-VJlO Costs ot sinsjle vohmies r.inj^e from $4.50 to 58.50. 
OxW-x retcr-.'iKi- ro J.ir.i .uliUms uxluiiques are included in .Appendix C :. 
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REPORTING RESULTS 



In the first chapter, we indicated that reporting the results of the eval- 
uation IS an integral aspect of the job. There are two issues involved in 
providing evaiuatum feedback: To whom should it be given? How should 
it be presented? 

AudiMCM for EvaliMtloii 

If an outside evaluator is hired to assess a women's studies program, that 
person hjw an ethical responsibility to report results to whomever did the 
hiring. The client is the first audience, and that is not always the 
women s studies program staff. It may be a dean or other administrator. 
In those cases m which someone has commissioned an evaluation with 
the intent of killing the program, there is really no way program staff can 
avoid that person's getting the results first. They can, however, try to be 
simultanet)us recipients of the results so they are able to present their own 
interpretations. Because the evaluator owes the client the first hearing, it 
IS probably to the program's advantage to be the client. 

Besides the legal client, others should be informed of the evaluation 
results. Anyone who participated in the evaluation has a right to some 
feedback ak)ut its results. Students and potential students have an in- 
terest in what is found about a program, as do staff and faculty in related 
fields. Ctillege and university administrators are another audience for the 
results. In addition, if the program was developed with Federal, State, or 
foundation hinds, these agencies are especially interested in the results of 
an evaluation even if it is not (and it frequently is) mandated. Finally, 
program staff and evaluators might well want to share evaluation experi- 
ences, rnethods and findings with others involved with women's studies 
St) that knowledge can accumulate across programs. This process could 
involve repcms (essays or brief research notes) to the Women's Studies 
Neus/tffter (Box 3)4, Old Westbury, New York 11568), work with 
reKioniU affiliates of NWSA, and presentations at national conferences. 

■Joint (.;.mmittire ..n tvaludtu.n St.»nddrdj. SuzrvLrdi W Uu^Mum Es^mtum Prepared for the 
Lilv K.unaatK.n. thr NjtK.nJ S:,cntc hmndation. jnd the Nat.i.n.i! Institute Education 1978 
niimir»n{r.iph«rd. p 101 
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Copies of evaluation materials and findings sent to the NWSA national 
office would become available to others in the field." We also would like 
to receive materials so we can revise and expand this handbcK)k. 

The lessons learned from Marcia Guttentag's pioneering work in the 
decision-theoretic approach to evaluation apply in any discussion of 
audiences. The more decisionmakers (regardless of funding S4)urce) are 
» involved in setting the questions the evaluation is to address and in 
following the ongoing evaluation activities, the greater the likelihcxxJ the 
results will be used. While there are dilemma*) here (e^g., what happens if 
we involve everyKxJy and the results are all awful— is it curtains for 
women's studies.'), in the vast majority of cases, it is always better to 
share some of the design resptmsibility with others. 

Pr«s«ii9hg RmuIIs 

ResuUM;an be reported in many ways, ranging from a full-blown mono- 
graph with statistical data and computer printouts to an interactive 
seminar. What is important is to present the results in ways that meet the 
needs o\ the various audiences. Given the numbers and roles of people 
who might be interested, developing different mcxies of presentations for 
each might prove prohibitively expensive. In that case, it mighr be gcxxl 
to develop three pnxlucts — the full report, an executive summary, and a 
press release — and augment those with whatever special materials are 
affordable. 

No matter how the results are repc^rted, the following information 
should be included: 

• A prt)gram description (this can be very abbreviated if the audience 
consists solely ot people who know the program well); 

• The evaluation design and analytic techniques (written in nontech- 
nical language for nontechnical audiences); and 

• The results. 

Further, whether presented in writing or orally, the rep<.)rt should be 
in clear, jargt)n-free English. Clarity of expression is a sign of clarity of 
thought, and jargon usually muddies issues rather than clarifying them. 
As a physician warned a friend about other dt>ctt)rs, "If they can't tell you 
what's wrong in plain English, rhey probably don't understand them- 
ves. 

In Appendix A is an exct of an executive summary of an eval- 
uation. It presents and answt program managers' questions about a 
numK'f ot women's studies aru: other prt^grams in a direct and concise 
way. 

NX'ithin this handKK>k, we have si^ught to provide information on 
eva!u<Kion ot wv>men*s studies in a clear, concise manner. Two other 
cippendixes. Abstracts of Studies and Res()urce Materials, repi^rt on 
results ot prevu>us wi>rk. 

\ I niNcrMfN ot M.»f%LinJ I t'llcj^t* Pari. ( ulk^^'f Wuk. M.irNl.irul 20742 




APPENDIX A: EXAMPLE 
OF EXECUTIVE SUMMARY 




PROJECT WELD 

WOMEN'S EDUCATIONi LEARNING AND DOING 

Funded by: The Fund for the Improvement of Postsecondary Educati 
the Department of Health, Education, and Welfare 

rroject Director: Margaret A. Talburtt, 

Formative Evaluation Research Associates (FERA) 
2 16 East Huron Street 
Ann Arbor, Michigan 48104 



Introduction 

This project was undertaken to provide information about three types of 
educational options available to undergraduate women: internships, 
women s studies classes. anJ skills development classes (or workshops). 
An internship was defined aV a collpge-sponsi)red program which had a 
stated goal ot career exptisure in an out-of-classroorti setting, such as a 
summer job with a federal agency. A women's studies experience was 
defined as an academic course which fcKused on women's contributions 
or relationships to a particular field, such as feminist politics or women in 
histor>-. Courses included in this group were not necessarily part of a 
formal women's studies pn>gram. The third category, skills development, 
signified a clas* or workshop which held the stated goal of behavioral 
cliange. Included were assertiveness training classes, career planning 
seminars, and personal growth classes. While these three program types 
did nor exhaust the many options that have recently been designed to 
foster uridergraduates* self-awareness, perst)nal growth, and professional 
p*>renrial. these programs represented some of the more pervasive and 
successful of the recent innovations. The goal of this study was to eval- 
uate the ourccmies of these programs. 



Method 



Within this framework, ei^jht schcx^Ls were chosen for participation in 
this study. Six of the eijijhi schcx)ls were predominantly women's colleges. 
The other two were women's centers at large coeducational universities. 
The rationale behind this choice was to begin an assessment of women's 
-ihkc at t onal -p n^g rr ss in se tt ings wlucli had commtued r hewiselves almost 
exclusively to the full development of women students. The eight schools 
represent a variety of institutions for women and formed a broad sample 
from which conclusions aK)ur women's education could be drawn. The 
Participating schcx)ls or programs were: Wellesley College, Mt. Holyoke 
yollege, Cedar Crest College, Wells College, Westbrook College, Mt. 
Vemon College, Everywoman's Center at the University of Massachu- 
sVtts, and The Women's Center at the University of Minnesota. 

xHie study began with preliminary visits to the programs at each 
schcxirJk^ring the fall of 1975. Of the eight schools, six had internships, 
seven hadWHR^en's studies courses, and three had skills classes. A total of 
270 students, W faculty and administrators, and 25 clients (or intern 
employers) were perstmally interviewed during the winter of 1976, using 
questionnaires designed for this study. Students were randomly selected 
from lists ot those who had completed their involvement with the pro- 
gram approximately one year prior to their interview. The majority of 
students were 19-23 years of age; fewer than 20 of the women were 30 
years ot age or older. 

Information was analyzed during the summer of 1976 and two re-' 
ports were sent to each program to contribute to their program 
improvement. In this way, some measure of formative evaluation was 
possible. The major crosS'Schcx>l report was completed during the winter 
of 1977. 

In addition, an advist^ry K)ard was formed to review project prog- 
re.^s, stimulate questions for analysis, and suggest applications of the 
project's findings. Members were Dr. Elizabeth Tidball of George 
Washington University, Dr. Sherry Penney of Yale University, Dr. 
Mariam Chamberlain of the Ford Foundation, and Ms. Carol Stoel of the 
Fund for the Improvement of Postsecondary Education. Their ideas, as 
well as thixse identified by the prof^nim directors, and the FERA staff, led 
tt> analysis of the for.cm ing points. 



Why did stiKtonts choos« th«se oxperiencf s? 

Summary: Students chose these various experiences to gain the general 
i>utcomes promised by each. Sixty-nine percent of the interns (n= 128) 
S4>ught career exposure, while 66 percent of the women's studies students 
(n = 82) desired new academic perspectives aK>ut women. Forty-four 
percent of the students in skills classes (n= 55) wanted new skills, such 



as assertiveness or communication skills, but another 40 percent' men- 
tioned new ideas or new attitudes as additional expectations. 

mMt dM tiMMi varlouft pro«niiiw 
oiifto>tud»nUt 

Suimmaryi Students reported that the three types of programs achieved 
the goals that they pa>mised. Internships promoted professional skills and 
career exposures. Women s studies classes fostered feminist perspectives 
and expanded concepts about what women can achieve. Skills classes led 
to increased self-understanding, assertiveness, and other skills related to 
pers».>nal effectiveness. 

Quantitative student data gathered with standardized forms, how- 
ever, also indicated that these various options achieved other important 
results whkh were not expected but were higWy valued by students. 
Further, their programs accomplished most of these outcomes signifi- 
cantly better than the traditional (or general) curriculum at each school. 
roT example, internships did more than provide career exposures. They 
increased self-confidence, openness, and assertiveness. Wo- -n's studies 
fostered selt-confidence and a greater sense of personal potential. Skills 
classes affected selt-contidence, independence, and feminist perspectives 
as well as interpersonal skills. In sum, each type of program accomplished 
ditterent results, and all programs resulted in some outcomes that 
traditional educational experiences did not produce. Simply, innovative 
structures have impact on outcomes that students rate as important— 
they do make a difference, supplementing the traditional experience in 
some important ways. 



What mrm thm dihmrmnemm b«twMn siiigl««s«x 
coHagas and coad univartitias? 

Summary: Because this study did not include a representative or large 
sample ot the varieties of especially coeducational schools, no firm con- 
clusions can be drawn aKiut differences among schot^l types. It would 
appear, how<>ver, that special programs for women enhance the quality of 
education that a female undergraduate at a coeducational university re- 
ceives, and may be even more important to her full development than 
similar programs at a women's college. Much more research needs to be 
conducted before such an inference can be accepted. 

What is tha impact off thasa programs? 

Summary: All three tyr^es of programs had impact on undergraduates, 
and substantially more impact than other educational experiences. 
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Eighty-four percent ot the interns and 85 percent of the students in skills 
classes said these experiences had ''some** to ''lots more** impact on them 
than their other courses and academic activities, whereas only 72 percent 
ot the students in women*s studies offered such ratings. Seemingly, the 
unique instructional processes and/or setting of the former explains the 
differential scoring. Nonetheless, the power of all three programs was 
i>bvii)us. 



W«r« students satisffM with tli^M •xp«ri«nc«s 
and what improvamants wauM thay afffar? 

Summary: Satisfaction with the process of the experience and with 
personal development was highest in the internships and skills classes, 
while satisfaction with academic content was slightly higher in the 
women's studies classes. Although this is not the way it has to be, it 
seemed that traditional instructional modes convey content better than 
innovative processes, while the nontraditional approaches seem to pro- 
mote greater personal growth. 

Despite the high satisfaction ratings, students did offer suggestions 
for improvement. In each type of experie.tce, stnicture was the most 
often mentioned type of desired improvement. More academic content 
was the most frequently mentioned addition that students desired. There 
was s<.>me evidence that students may not know how to evaluate innova- 
tive educational experiences because they do not know exactly what to 
expect from them. Indeed, more than 75 percent of each group of stu- 
dents replied that positive unexpected outcomes had occurred in their 
respective programs. 



Did thasa axpariancas ralata ta 
studants' ffutura ralas? 

Summary: Yes! Although a total of eight types of roles were mentioned, 
the major roles that students expected to assume are family and career. 
The most imp^^rtant skills they felt that they would need to be successful 
in those roles were self-confidence, interpersonal skills, and technical/ 
profes^siofial skills. Clearly, these are the types of skills and attitudes that 
the three innovative educational experiences fostered. Within the 
^groups» the interns were the most career oriented and the women*s 
Ntudies students the least family oriented. But to pretend that young 
women tcviay do not seek family as well as career roles would be an 
erroneous assumption. 
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Did tlMM studMits fMl tiMir Instnutlons eouM h% 
improvMl tehools for wmiMnt 

Summaryt Ajjain. yes! Seventy-six percent of the women at single-sex 
schiK)ls and 57 percent of the women at coeducational universities felt 
that their colleges could be improved as schools for women. Academics, 
support services, and structural improvements were suggested as focuses 
tor change. 



Conclusion 

This abstract contains many highlights of a full report. Specifically, use- 
tul data from the faculty interviews as well as more extensive student 
information are missing from this summary, The implications of these 
findings are many, however, and should be of interest to the students, 
faculty, program directors, and sponst)rs of programs for women. All of 
these groups should recognize the specific benefits of each type of pro- 
gram and relate these to their educational purposes in an integrated way. 
Clearly, each ot these programs makes an important and unique contri- 
bution to the education of women and one that is not available in 
traditional higher educational settings. 
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APPENDIX B: 
ABSTRACTS OP STUDIES 



IXKiiments with an ED number are contained in the Educational Re- 
sources Information Center (ERIC) collection. Microfiche or paper 
a>pies ot these diKuments can be obtained for a fee from the ERIC 
IXicument Reproduction SerNMce (EDRS), P.O. Box 190, Arlington, 
V ir0nia 222 10. 



Title: "Assessment of the Experi^BCnlCeaming for College Credit 

in the Area of Women Studies" 
Authors: Benjamin Sackmary and Hannah Hedrick 
Source: Educational Resources Information Center, ED 1 55 208* 

The paper prop*.>ses a general set of guidelines for grantirig experiential, 
or lite, learning credits. Examples from portfolios of women applying for 
women's studies credits based on their life experiences are presented, but 
there is no indication of whether — or how many — credits were granted. 
There are. however, samples of women's studies courses and goals from 
i I colleges that may prove useful. 



Title: "Attitudes of Students, Faculty, and Administrators 

Toward a University Women's Studies Program" 
Authors: Glen L. DeBiasi and Carolyn RkxJes 
Source: Old Dtwinion University, Norfolk, Virginia 23508 

The study involved an investigation of attitudes toward a new wt)men's 
>tudie.s program held by nonparticipant students, faculty, and adminis- 
trators. A representiitive sample of each group was administered ques- 
tionnaires concerning the awareness of and attitudes toward the program 
mk\ Its goals. Results showed that a relatively large percentage of the 
rosp^indents were aware of the women's studies program, but fewer were 
aware of its specific goals. A majority were supp^irtive of the program. A 
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relatively larjje percentage of faculty (60 percent) and administrators 
considered women's studies a valid field of academic inquiry. 



Tide: ' -trtects of Women's Studies C \>urses on Worwn's 

Attitudes and Goals" 
.Atttheri:--Et4ernMr<:;arnY "" """ " 

Source: Educational Res^nirces lnformatit)U Center, ED 1 50 4W* 

C'anty used a pre-, post design with a comparison group. The experi- 
mental group consisted of students in the Psychology of Women and 
Psychology ot Adolescence courses; comparison group students were 
cnrt)lled in Psychological Statistics. The measures were the Spence and 
Helmreich Attitudes Toward Women Scale, the California PsvchoK>gical 
hn entory, Broverman's Sex Role Stereotype Questionnaire, and a de- 
tailed locally deveK>ped questionnaire. The results were that thg ex- 
perimental group had more liberal plans to combine marriage and career 
and to delay marriage. Further, there were changes in attitudes toward 
women^s vocational, educational, and intellectual roles and toward 
marital relationships 



Title: "Evaluating Women\s Studies: 

A IX'Cision-Theoretic Approach 
Authors: Marcia Cjuttentag, Lorelei R. Brush, Alice Ross Gold. 

Mamie W. Mueller, Sheila Tobias, and 

Mami Goldstein White 
Source: Si^jris, VoL ^ No. 4, 

The article presents an introduction to the Multi- Attribute Utility 
Pec i.Mon 'Theoretic (MAUT) model for conducting evaluation. The 
authors argue that the MAUT approach to evaluation is more effective 
tor evaluating women's studies than traditional approaches for four 
reasons. First, the MALT model addresses such questions as, **How can 
the program be improved.*'' and "Hovy can it best meet the program goals 
ot students, faculty, and administrators.'*' Second, the model allows dif- 
ferent subgroups within a program to develop their own set of goals by 
which thev can judge program success. Third, the MAUT mcxlel enables 
program personnel to chtmge goals and/or me«uis of achieving them at 
any time during the ovaluatu)n process. Finally, it allows for continuous 
oN ahianon ot a program. 

There are three major steps m the use .1 the MAUT model: **Stat- 
the program's ^luiK m the order ot their priority and assessing the 
probability ot acl\ieving them, developing techniques to measure the 
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degree to which the program achieves those goals, and collecting and 
analyzing data generated by the administration of these measures. " 

The following steps are involved in stating priorities. 

1 . Identify all groups whose goals for the program should be evaluated. 

2. Identify the issues to be addressed by the evaluation. 

3. Identify the action alternatives to be evaluated. 

4. Specify the goals of each group identified in the first step, 

5. Rank the goals in priority order. Each constituency that has specified 
goals must agree on their importance. 

6. Assign '^importance weights" to each goal. 

7. "Sum the importance weights, divide each ... by the sum, and 
multipfy by 100." 

8. Judge the probability that each alternative can achieve each goal. 

^. Calculate the total utility of each action alternative (the sum of the 
importance weight times the probability of success). 
Measurement of goal achievement in the MAUT model conforms to 
normal standards of data gathering. Baysian statistics are used for data 
analyses. 

V 



Title: ''An Evaluation of a Women's Studies Program'' 
Authors: Sharon A. Shueman and William E. Sedlacek 
Source: Joumcil ofNAWDAC. Fall 1977 

The authors repi>rt on evaluation of the women's studies program at the 
University of Maryland, Q)llege Park. The evaluation emphasized input 
variables (the structure of the program and a description of the students) 
and output variables (short-term effects of the courses on attitudes and 
selt-concepts). A comparistm group was used. The measure included a 
questionnaire on student demographics, goals, and experience, one on 
student attitudes toward the course, the Bem Sex-Role Inventory, and 
the Tennessee Self-Concept Scale. The last iwo were used as pre- and 
post-tests. Significant changes were found on items that indicated 
ciwarcnes6 ot sex roles but on no other items. 



Title: "Final Evaluation of OCR 66: I mages ot Women m 

Literature. University of Minnesota** 
Authors: Patricia l>avis et al. 

Source: Educational Resources Information Center, ED 124 240* 

l>avis and her associates recinmt the histor>' oi the course and its eval- 
uation the third time it was offered. The history includes a discussion of 
the development of the syllabus as wt*ll as of the evaluatiim. Instructors 
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tor the course were primarily responsible tor the evaluation but received 
assistance from the University's Research Center. 

In Jevelopin^ the evaluation plan, the instructors clarified their 
yoals and objectives, The underlying evaluation questions were: [I) Did 
the course meet us objectives' and (2) Were the students different from a 
comparison yroup (those enrolled m **Man s Religious Beliefs?''). Using a 
pre-, post-design, students were tested on their knowledge of women in 
literature, the histor>' of the women's movement, and the contributions 
ot women to society as well as their attitudes toward women. The instru- 
ments testing the knowledge objectives were developed locally, and a 
standardized test was used to test attitudes toward women. In addition, 
an open-ended questioniiaire was developed to ascertain students' expec- 
tations ot the course. 

The tindings were generally positive and related to instruction. 



Title: "Nursing; A Feminist Perspective'* 
Authon: Phyllis Kritek and Laune Glass 
Source: Nursmni^alDok, March W78 

The authors recount the development of a course that was cross-listed in 
the ?>ch(K)l ot nursing and the office of women's studies. The course was 
designed to bring a feminist perspective to bear on the nursing profession. 
Topics included: histor> ot the women's movement as related to nursing; 
nurses a> women (socialization, ptwer, politics); sexism in health care; 
strategies ti)r the future. The authors developed their own evaluation of 
the course. The evtiluation mcluded an iissessment of student learning 
(examinations ar(d evaluatum ot research projects), the Allport Study of 
V alues. a self-deyvlttped e\>4luation tu)l, and student-written evaluations 
ot films, iectures^^ discuksions» and the instructors' performance. Results 
indicated that stiklents valued the course, believed they learned some- 
thing from it. and met their personal goals. G)urse evaluations were 
[^>sirive. The authors have used suggestions frt)m the evaluations for 
modifying and impruvini; the course. 



Title: "The Paradi^x ot intention and Effect: A Women's 

Studies CJoursc" 
Authors; Lorelei R. Bru^h. Alice RossCiold. and 

Mtirni C joldstein VC'hitc 
Source: Sij^is, Vol. ^ No. 4, 1^78 

The aut[u)rs report a comprehensive, long-term evaluation of "the 
imp<icr of an inter-discipliHiiry women's studies course on the sell- 
o>rKeprs. sex role attitudes, and sex-role stereotypes of the students 
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based on data gathered over a two-yfar period at a small liberal arts 
college." Two methods of data collection were used: an open-ended 
in-depth interview and pre-post administration of a questionnaire and of 
a battery of tests (e.g., Minnesota Women's scale, Brovermari Sex-Role 
Stereotype Scale, the Bern Sex-Role Inventory). Students in other 
courses served as comparisons. The authors found that the in-depth 
interviews yielded more interesting and worthwhile data- Results indicate 
that while some students were deeply affected by the course, most 
changes were small and varied from year to year. Students did become 
more confident and able to defend their beliefs with information. 



Title: "The Reported Impact of Women's Studies Courses 
on Students' Lives" 
- Authors: Allana Cummings Elovson and Irene Cockroft 
Source: Educational Resources Information Center, ED 160 944* 

Elovson and Cockroft describe an innovative approach to measuring the 
impact of women's studies courses on the lives of students— a type of 
goal-free evaluation. They developed an open-ended survey instalment 
with 23 questions that asked students to report the importance and 
impact of women's courses on their lives. The majcT innovation was the 
development of a scoring system that allows the coding of responses 
regardless of the specific questions that triggered them. Ninety-three 
percent of those surveyed reported positive impacts, including: 

1 . Increased confidence in and awareness of abilities; 

2. Higher career aspirations; 

3. Broadened perspectives on history and social issues; 

4. Increased comprehension of political,, economic, and social factors in 
shaping human experience; 

5. Greater tolerance of those with different histories and value systems; 

6. Increased desire to participate in and contribute to the study; 

7. More questioning, analytical, and rational attitudesi 

8. Less acceptance of exploitation and manipulation. 



Title: Seven Years Later. Women's Studies Programs in 1976 
Author: Florence Howe 

Source: National Advist^ry Council on Women's Educational 

Programs, 1832 M Street, N.W., Washington, D.C, 20036 

The review ot women's studies programs was shaped by the following 
questions. 

I. What is the current state of mature women's studies programs with 
respect to their faculties, students, curricula and classrooms, their 
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internal governance structures and administrative relationships to the 
universities in which they exist ' 
2. What has been the impact of these programs on their campuses and In 
their wider communities? 

What dominant issues and needs are critical to the future of women's 
studies' 

Howe tound that enrollments in women's studies weVe growing, that ^ 
a wi !e variety and depth of courses were being offered, and that there is a V^, 
**growing realization of institutional impact/' 

Hi)we concludes: 

• Thai the tuture ot women's studies will depend K)th on the ability of 
Its advocates to mainta^i^n thetr present high levels of energy, commit- 
ment and direction, and on the ability of the institutions to make the 
accommtKiations required not only by student?), but also by faculty and 
programs; 

• That while the energy commitment and direction of the advocates are 
abundantly present no\v» there are many who seek Si)me stronger indi- 
cators that the productive and exciting level of women's studies 
teaching and scht^larship is being provided or will be provided in the 
future, with commensurate institutional rewards. 



Tide: 'Women in Continuing Education" 

Authors: Linda M. Rosenwcx>d and Patricia Lunneborg 

Source: Educatumal Resources Information Center, ED 067 997* 

The authi^rs sought to find the effects ot continuing education on 
wonien\ seh'image» problem-solving abilities, and career orientation. 
Particip^ints in short, career-oriented workshops who later signed up for 
courses were compared to those who did not. The instruments were 
developed by the evaltiators tor the affective measures, and a pt^riion o\ 
the L'niversity of Washington entrance exam was used to test problem- 
solving abilities. The only significant positive change was in self-esteem. 
Although the authors cire some methtxiological flaws in rheir study, they 
suggest that the program should reconsider K)th its gtnds and methixls. 



Tide: ^'NX'onien's Studies as C-hange Agent" 

Authors: Ruth Scotr, .^nn Richards, and Marie Wade 

Source: Psychnlnffy n/ VX'umt'n Quarterly, Vol. 1, N(^. 4, Summer 1977 

The authors repi)rt a study of changes in attitudes toward wc^men by those 
ciuolled in women's studies courses. Participants in the study were en- 
roil«;d \n women's studies courses <it a bberal arts college and at a legiona! 
can^^'us ot A state uni\'ersitv. A comparison group consisted ot psychology 
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students .It the lik>Ml .jrrs coIIokl-. The research instrument was the 
.Spcnce and HelinreK h Attitudes Toward Women Scale. Results showed 
that attitiide.s toward wonien became more liberal after takin« a women's 
studies course while there wa> ru) chani^e in the comparison ynnip's 
attitude. 



Title: "VComen's Studies .it I 'tab State L-'niversity: A Proposal" 
Author: judith M. Cl.ipp.i 

Source: Hducitmnal Resources Intormatiun Center, ED 145 822* 

The author re{\)rt> in detail the process ot developing a proposal tor a 
women's studies program. This pri)cess be«an with an on-campus needs 
av^essment and moved to a proposal tor a full women's studies proj^ram. 
The propoNal included reconitnended evaluation procedures. 

t Mppa surveyed faculty .ind students to determine how they viewed 
changes m roles tor the tuture and the ettectiveness ot the curriculum 
tor prepanny students tor tfiose roles. She also developed a complete list 
ot woiiK-n's studies courses that were avail.ible or bein^ developed on 
campus. .1 |,>f ot courses that could be revised to include a women's 
studies coiuf^'nent; and independent studies, re.search. and tieldw.,)rk 
opportiHuties relatifii; to wonu-tv 

TIk- next sref^ in her process w.is to develop yoals and objectives tor 
a total pr*>i;rain- -and the relevant evaluative .ipproach, The remainder 
ot this siimni.irv lists some ot tliose ^oAs and objectives wiih examples ot 
Ci.ipp.i's ev.iluation indic.itors. 

CiOAl I. 

i Vvelop .1 suttiLieru number ot courses so scu.Jents have the options ot 
cUvtiieN or .in wTiM ^t[Jle^ o -ru i nfr,iti,m n/ erri^',asis within lUx'ral arts. 
Objectis es 

.A. C oordin.ife exisruii: ottenn^s to avoid duplication. 
Antw (/Mfi'tl iiuK onieN 

I Tot.il ruiniber ot courses littered each academic year will be 
eveiiK distributed .i.ross quarters ,ind will not meet at the s.inie 
fiiiie, 

bnrollment u ill wk rease. 
ri.Koiiot' I -e a torni (h,it can be completed from rhe catalog and 
iiK liides .1 >p,ice to note unanticipated outci'mes. 
fV I V\ elo[^ ,!iui impletnenr procedures tor acceptini; new courses. 

■AfUivi/Mfi'd miUi lines 

I Proptised Courses will follow rhe procedures ot the Jeparrmenr. 

>.ollei:e, ,ind edui. .itional policy committee. 
- 1 he Women's studies coordinator will .id\ ise and assist. 
/'''kcJ;(7c I se .1 worksheet tli.it includes columivs tor content, in- 
srriktion.ii st\le, student ev ilu.ition, and tacultv comments. 
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C. Provide a variety ot learning experiences for students. 

Anticipated imtanru:: 12-15 departments will offer 50-60 credits in 

Women's Studies. 
PuKedure Use the 5jame worksheet as in B. 

GOAL 11: 

Support efforts in A\ departments to improve curriculum with respect t<^ 

women so that objective, nonstereotyped material is presented. 
Objectives 

A . I nc rease researc h . 

B. Disseminate material. 

C. Undergo careful department evaluation ot the quality ot research and 

curriculun> offermj^s. 

CiOAL 111: 

Assist the Women\ Center. 
Cioal IV: 

Bncourajkje participation ot men. 

(The la.Nt three ^jiials included the same categories ot anticipated out- 
comcN and evidiiation procedures as the tirst two. ) 

The article includes an excellent annotated biblioj^raphy. 
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APPENDIX C: 
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